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NDER THE ROBERT WOOD JOHNSON FOUNDATION 
: | (RWJF) Program on Chronic Mental Illness (PCMI), cities ex- 
tensively reorganized their services for the target population. 
The logic of this intervention derived from the premise that fragmen- 
tation of administrative, fiscal, and clinical responsibilities is a major 
impediment to effective services for persons with CMI. Model clinical 
programs addressing this fragmentation have been found to improve 
continuity of care and outcomes (Aiken, Somers, and Shore 1986; Shore 
and Cohen 1990; see also the article by Goldman, Morrissey, and 
Ridgely in this issue). The RWJF intervention itself occurred at a high 
administrative level by establishing citywide local mental health authori- 
ties. These authorities sought to improve services and continuity of care 
at the provider level for all persons with CMI by centralizing administra- 
tive, fiscal, and clinical responsibility. This improvement in services at 
the system/ provider level was expected to translate into improvements in 
services at the client level, leading in turn to better client outcomes. 
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Based upon this program logic, the following evaluation questions 
were addressed in this client outcome study: 


1. Did continuity of care and case management improve during the 
course of the program? 
2. Did client outcomes improve during the course of the program? 


Because persons with CMI are highly diverse and because, inevitably, 
client subgroups vary in their exposure to aspects of the program, a study 
of its impact on treatment and outcome at the client level can be carried 
out more efficiently by focusing on the targeted subgroups most likely to 
be affected by the program. This focus, analogous to a “tracer condition” 
in medical studies, will not provide information on other subgroups of 
clients; however, it will tell us whether or not groups of clients viewed as 
highly likely to be affected by the program indeed experienced improve- 
ments in services and outcomes. The observation of program effects would 
lend support to the hypothesis that the program influenced the system 
of services for persons with CMI, and at the very least would inform us 
of such effects among high-priority subgroups of clients. The absence of 
program effects among highly targeted subgroups of clients with CMI 
would raise questions about whether the program had any significant im- 
pact at the client level regardless of effects observed at the system level. 

A variety of subgroups of high-priority clients with CMI can be iden- 
tified: those who are homeless, unserved, in crisis, of in some critical 
transition (such as hospital discharge or move to a new residence). As a 
general strategy, we chose to evaluate the program impact among persons 
with CMI who were at the point of discharge from intensive 24-hour care 
(short-term hospitalization) because they are at a point of high need and 
vulnerability to system failures, are likely to be affected by the types of 
system changes foreseen under the RWJF program, and do not present 
the extreme methodological problems of case identification and tracking 
posed by certain other high-priority CMI groups. 

In addition, we decided to focus on continuity of care and case man- 
agement as the key client-level service outputs because of the central role 
these concepts played in the development of the RWJF program (Shore 
and Cohen 1990) and their prominence in current views of what consti- 
tutes good care for persons with CMI and what typically is lacking in ex- 
isting service systems (Bachrach 1981; Aiken, Somers, and Shore 1986; 
Lehman 1988). 
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Methods 


Overview 


This client study utilized a quasi-experimental design with replications 
in four of the nine RWJF demonstration cities: Baltimore, Cincinnati, 
Columbus, and Toledo. Two separate cohorts of individuals with CMI 
from each city participated in the study. Both sets of cohorts were in- 
terviewed at baseline (hospital discharge) and at two follow-up points: 
2 and 12 months after discharge. Data collection from the first cohort 
began in November 1988 and ended in June 1990. For the second co- 
hort, interviews were conducted from September 1990 to February 1992. 
The RWJF awards to initiate the local mental health authorities in the 
nine cities began in November 1986. Therefore, the first cohorts reflect 
service experiences early in the development of the authorities and are 
not true preprogram baselines. 

To obtain a reliably homogeneous sample of individuals with CMI, 
our screening process specified the criteria for selecting study partici- 
pants: subjects who had been hospitalized because of a mental disorder; 
who met the Ohio criteria for severe mental disability (based upon diag- 
nosis, duration of treatment, and disability)(available upon request from 
the authors); whose current hospitalization lasted no more than 120 days; 
who were between the ages of 18 and 64; who spoke English, resided in 
the study area, and were legally competent. A liaison assigned to each 
hospital identified the potential subjects and obtained the consent of ell- 
gible subjects to participate in the study. Trained interviewers contacted 
these clients and interviewed them initially within a defined time frame 
spanning from four days prior to hospital release to seven days following 
discharge. 

A structured interview was used to elicit respondents’ sociodemographic 
characteristics, current clinical status, level of functioning, quality-of-life 
judgments, mental health services received in the past, perceived psy- 
chosocial and clinical service needs, and information on whether these 
needs were being addressed. 


Subjects 


Among the large pool of clients experiencing acute 24-hour-care ept- 
sodes who were screened in the four cities (2,438 for Cohort 1 and 3,417 
for Cohort 2), 46.3 percent (1,127) in Cohort 1 and 36.3 percent (1,042) 
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in Cohort 2 met the study eligibility criteria. Among the eligible clients, 
359 (42 percent) agreed to participate in Cohort 1 and 302 (31 percent) 
agreed to participate in Cohort 2. Compared with Cohort 1, Cohort 2 
refusal rates were lower (37 percent versus 19 percent, respectively), but 
the rates of loss due to discharge prior to study recruitment attempts were 
higher (22 percent versus 50 percent). Although consent rates were con- 
sistently higher for the cohorts in the three Ohio cities (CIN = 40 per- 
cent, COL = 40 percent, TOL = 42 percent) than for those in Baltimore 
(28 percent), consent rates did not differ significantly by diagnosis for 
both cohorts across cities (schizophrenia = 39 percent, other = 43 per- 
cent). Consenters did not differ from the entire pool of eligible subjects 
on gender, age, or race. 

Comparison of the characteristics of subjects recruited into the two 
sets of cohorts revealed that the two groups had similar demographic 
characteristics, levels of functioning, and clinical status at baseline. The 
mean ages of the subjects were 35.1 for Cohort 1 and 36.4 for Cohort 2 
(n.s.). Nearly half of the subjects from both cohorts were women (47.9 
percent and 44.7 percent, respectively, n.s.). Almost half of each group 
was Caucasian (47.1 percent and 44.7 percent, respectively, n.s.), and 
the majority of both had never been married (58.6 percent and 52.8 per- 
cent, respectively, n.s.). In terms of self-rated level of functioning, the 
two cohorts were equivalent (scale: 1 = excellent, 4 = poor, both cohort 
means = 2.6). A higher proportion of Cohort 1 was employed prior to 
admission (21.8 percent versus 16.3 percent), but this difference was not 
statistically significant. 

More than three-fifths of participants from both groups (63.1 percent 
of Cohort 1 and 61.9 percent of Cohort 2) had a primary diagnosis of 
schizophrenia. Both groups had spent on average over 50 days in the 
hospital during the past year before the first interview (51.3 days for 
Cohort 1 and 54.6 for Cohort 2). The mean number of hospital admis- 
sions during the preceding year for the two cohorts also did not differ 
(Cohort 1 = 2.1, Cohort 2 = 2.3, n.s.). However, statistically, although 
probably not clinically, significant differences in baseline symptomatol- 
ogy were found between members of the two cohorts (symptom checklist 
[SCL] scales:1 = none, 5 = extreme; Cohort 1 mean = 2.3 [s.d. = 0.89]. 
Cohort 2 mean = 2.2 [s.d. = 0.84], P < .04). 

Very high retention rates for both cohorts were attained at the 2-month 
and 12-month follow-up points, with completion rates ranging from 
86 percent to 97 percent. 


Continuity of Care and Client Outcomes 109 


Method of Analysis 


The objective of the analysis was to test whether differences existed be- 
tween the two cohorts on continuity of care, case management, and out- 
comes at the two follow-up points after hospital discharge. In order to 
relate any observed between-cohort differences in the impact of the RWJF 
program, it was necessary to control for potentially confounding differ- 
ences between cohorts at baseline, such as the baseline value for the de- 
pendent measures of interest, demographic characteristics (age, gender, 
education, and marital status), diagnosis, and city. Analysis of covari- 
ance (ANCOVA) was conducted to partial out the confounding variables 
by treating them as covariates. Although comparison of the patients from 
the two cohorts on baseline demographic characteristics and diagnosis 
upon entry to the hospital did not show significant differences, partial- 
ing out their effects reduces error variance. In addition, a city-by-cohort 
interaction was included in the analysis to detect and control for any 
treatment-by-site variations. An alternative series of ANCOVAs was con- 
ducted using a composite baseline covariate derived from a combination 
of variables (demographics, diagnosis, prior hospitalizations, symptoms, 
and objective and subjective quality of life) that best predicted cohort 
membership in a logistic regression. Results from these analyses yielded 
the same results that we have presented here. 

After partialing out these confounding variables, the main effect of 
cohort could be more accurately assessed. It was then possible to consider 
each test as a one-factor (cohort) analysis of variance on a dependent 
variable adjusted for the covariates. Cohort differences were assessed sep- 
arately at the 2- and 12-month follow-up points to determine if there 
were significant cohort effects at either one or both time points. 

The first evaluation question asked whether case management and 
continuity of care changed over time (between cohorts). Five variables 
reflecting certain dimensions of continuity of care and case management 
were analyzed, and included the following: 


1. whether the respondent had a case manager during the interim 
assessment period 

2. whether clients with case managers had experienced a change in 
their managers during the interim period 

3. whether the respondent experienced any service needs during the 
interim period 
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4. how many of these needs were addressed by services during the 
interim 
5. the respondent’s rating of the helpfulness of the services 


We expected that if the RWJF program was effective in improving 
continuity of care, Cohort 2 participants would be more likely to have 
case Managers, to have greater continuity in the identities of these case 
managers, to identify fewer service needs, to report a greater proportion 
of service needs addressed, and to rate the helpfulness of their services 
more highly than individuals in Cohort 1. 

The second question concerned differences in outcomes between co- 
horts. A large number of outcome variables were assessed using the Leh- 
man Quality of Life Interview (Lehman 1988), adaptations of subscales 
from the Uniform Client Data Instrument (Goldstrom and Manderscheid 
1986), and the Tessler Continuity of Care Provider Questionnaire (Tess- 
ler, Willis, and Gulman 1986), the Denver Consumer Questionnaire 
(Ciarlo and Rethman 1977), and the SCL-90 (Derogatis 1983). Because 
the findings were highly consistent across all of these measures, we will 
present the results from only a few key summary measures. Results from 
two subjective quality-of-life variables— general life satisfaction and sat- 
isfaction with living situation (Lehman, Ward, and Linn 1982)—and 
three objective outcome indicators— number of nights hospitalized per 
month during the study period, symptomatology (as rated by three sub- 
scales of the SCL-90) (Derogatis 1983), and self-reported level of func- 
tioning (Goldstrom and Manderscheid 1986)— will be reported. In our 
design, significant improvements in outcomes related to the program 
would be evidenced by Cohort 2 achieving better scores on these vari- 
ables compared with Cohort 1. 

For variables that obtained a significant (P < .05) main or interaction 
effect, adjusted means were examined to determine the direction of the 
effect. 


Results 


Case Management and Continuity of Care 


Results of the ANCOVA performed on whether the subjects had a case 
manager indicated a significant cohort main effect at 2 months’ follow- 
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up (F = 12.16, df = 1, 563; P< .001). By 12 months’ follow-up, this 
significant cohort main effect disappeared, but a significant city x co- 
hort interaction effect emerged (F = 6.67, df = 3, 537; P< .001). Fur- 
ther analysis on the cohort-adjusted means (table 1) revealed that a 
significantly higher proportion of Cohort 2 subjects received case man- 
agement services (mean [M] = 0.76 versus M = 0.64; ¢ = 3.30; P< 
.001) during the 2-month period after hospitalization. This trend was 
significant in Cincinnati, Toledo, and Columbus. There was no such 
trend in Baltimore. At the 12-month follow-up point, the significant 
difference between the two cohorts had disappeared in the overall com- 


TABLE 1 
Case Management Services Received: City < Cohort-adjusted Means 


Mean? differences and probabilities 


Two months after baseline Twelve months after baseline 
City Cohort 1 Cohort 2 Difference Cohort 1 Cohort 2 Difference 
Have a case manager (%) 
Baltimore 42 40 —0.02 52 38 —0.14* 
Cincinnati 70 83 0.13** 84 80 —0.04 
Columbus 74 92 0.18* 64 92 0.28*** 
Toledo 70 88 0.18* 80 78 —0.02 
All cities 64 76 0.12*** 70 72 0.02 
Changed case managers (%) 
Baltimore 24 27 0.03 >») 17 —0.38* 
Cincinnati 15 5 —0.10 49 31 —0.18* 
Columbus 7 12 0.05 37 29 —0.08 
Toledo 25 10 —0.15 48 33 —0.15 
All cities 18 14 —0.04 47 27 —0.20** 


Number of non-case-management 
services received 


Baltimore 3.73 3.27. —0.46 4.21 3.36 —0.85** 
Cincinnati 3.28 3.54 0.26 4.05 3.66 —0.39 
Columbus 3.43 3.27. —0.16 3.57 3.79 0.22 
Toledo 3.58 3.31 —0.27 3.05 3.46 0.41 
All cities 3.51 3.35 —0.16 3.72 3.57. —0.15 


4 Adjusted for baseline lagged variable and subject characteristics. 
* P< 05,7 2 <0l, P<. 001. 


aii a 


112 Anthony F. Lehman et al. 


parison, but remained significant in the positive direction in Columbus. 
Baltimore, on the other hand, demonstrated a significant decrease in the 
percentage of clients who had a case manager over the 12 months. 
Whether respondents had changed case managers—an indicator of 
continuity of care—was examined next. A lower propensity to change 
case managers indicates better continuity of care among these subjects. 
The ANCOVA yielded a significant cohort main effect (F = 7.96, df = 1, 
288; P < .01) 12 months after hospitalization. Table 2 shows the 
direction of this significant main effect: a greater percentage of Cohort 1 
(M = 0.47) changed case managers, compared with Cohort 2 (M = 0.27) 
during the one-year period. Although all four cities exhibited this pattern, 


TABLE 2 
Needs Addressed and Helpfulness of Services: City x Cohort-adjusted Means 


Mean? differences and probabilities 


Two months after baseline Twelve months after baseline 
City Cohort 1 Cohort 2 Difference Cohort 1 Cohort 2 Difference 
Needs addressed/needs identified 
Baltimore 79 .89 0.10 85 85 0.00 
Cincinnati .74 73 —0.01 .82 .80 —0.02 
Columbus 85 74 -0.11 .88 82  —0.06 
Toledo 81 82 0.01 91 93 0.02 
All cities .80 79 —0.01 87 85 —0.02 
Have no needs (%) 
Baltimore 17 46 0.29*** 26 37 0.11 
Cincinnati 39 22 —0.17** 36 34 —0.02 
Columbus 19 27 0.08 26 25 —0.01 
Toledo 28 22 —0.06 69 26 —0.43*** 
All cities 26 29 0.03 39 30 —0.09* 
Helpfulness of services? 
Baltimore 1.73 1.75 0.02 1.71 162 —0.09 
Cincinnati 1.85 1.83 —0.02 1.90 1.87 —0.03 
Columbus 1.57 1.79 0.22 1.81 1.80 —0.01 
Toledo 1.68 1.72 0.04 La72 FatZ 0.00 
All cities 1.71 1.77 0.06 1.78 1.75 —0.03 


® Adjusted for baseline lagged variable and subject characteristics. 
b 1 = very helpful, 4 = not at all. 
* P< .05, ** P< .01, *** P< .001. 
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statistically significant results were noted only in Baltimore and Cin- 
cinnati. 

Results of the number of types of mental health care services, other 
than case management, received by the two cohorts suggested a nonsig- 
nificant cohort difference across cities for both study periods. However, 
there was a modest and significant city x cohort effect at the 12-month 
follow-up (F = 3.74, df = 3, 547; P< .05). Further analysis conducted 
on the city X cohort adjusted means (table 1) revealed a significant dif- 
ference in favor of Cohort 1 in Baltimore; that is, clients in Cohort 1 re- 
ceived a broader variety of service categories compared with Cohort 2. 
Such a finding indicates a decrease in the breadth of aftercare services 
obtained by the Cohort 2 subjects in Baltimore one year after discharge 
from the hospital. A similar but nonsignificant pattern was observed in 
Cincinnati for the same time period. 

It is beyond the scope of this article to offer detailed descriptions of 
changes observed in service categories other than case management. How- 
ever, a brief summary of changes in each city at the 12-month follow-up 
point is instructive because of the cross-city variations. The categories 
of services assessed included case management, psychosocial rehabilita- 
tion, housing, vocational rehabilitation, psychiatrist, and therapist. For 
each category, the dichotomous response was whether or not the client 
had received the service during the preceding 10-month period. 

In Baltimore there was a decrease in the proportion of clients report- 
ing receipt of case management services (53 percent in Cohort 1 versus 
37 percent in Cohort 2, P< .05) and housing services (76 percent versus 
42 percent, P< .001) 12 months after hospital discharge, but no signift- 
cant changes in other service categories. In Cincinnati, housing services 
(70 percent versus 55 percent, P < .05) decreased from Cohort 1 to Co- 
hort 2 at 12 months. In contrast, Columbus clients reported increases in 
case management (64 percent versus 91 percent, P < .001) and psychia- 
trist services (68 percent versus 90 percent, P < .01) between Cohorts 1 
and 2. Finally, in Toledo, a dramatic increase in psychosocial services was 
noted (13 percent for Cohort 1 versus 61 percent for Cohort 2, P< .001). 
Clearly service changes varied considerably across time in the four cities. 


Needs Addressed and Helpfulness 


of Services 


To determine how well the needs of clients were being addressed, the in- 
terviewers enumerated psychosocial and medical/clinical needs for the 
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respondents. The respondents were asked (a) what needs they had; and 
(b) if they had a need, whether they had received help for it. A ratio of 
the total needs addressed to the total needs identified (based upon mea- 
sures determined by Tessler, Willis, and Gubman 1986) was computed 
for each respondent. Results showed nonsignificant cohort and cohort x 
city effects. Similarly, a comparison of the needs-addressed to needs- 
identified ratio (table 2) revealed no significant overall difference be- 
tween the two cohorts at either study follow-up point. This overall trend 
of no difference held across all of the cities. 

A substantial number of respondents from each cohort did not have a 
need for the services enumerated in the interviews. Thus, a dichotomous 
variable that indicated whether a respondent had any needs was derived. 
ANCOVA was conducted using this variable as a dependent measure, and 
the results yielded a significant city < cohort interaction effect (F = 8.38, 
df = 1, 576; P< .001) in the 2-month follow-up period (table 2). The 
simple effects ¢-tests conducted on covariate-adjusted means in each city 
revealed that, in Baltimore, a larger percentage of Cohort 2 reported 
having no needs, compared with Cohort 1 (table 2). In Cincinnati, the 
reverse pattern was observed. During the second follow-up period (12 
months), significant cohort and city x cohort effects emerged. T-tests 
conducted on cohort-adjusted means indicated that a significant and 
lower overall percentage of respondents in Cohort 2, compared with Co- 
hort 1, claimed to have no needs (M = 0.30 versus M = 0.39, ¢ = —1.73; 
P< .05). A closer examination, however, revealed that this finding held 
only for Toledo (table 2). 

The analysis on the helpfulness of services revealed no significant dif- 
ferences between the two cohorts during follow-up (table 2). 


Changes in Outcome 


Of all five outcome variables, the only significant difference in outcomes 
between cohorts was observed at the 12-month follow-up on the symp- 
tom scores (F = 7.06, af = 1, 524; P< .01). A simple effects f-test 
revealed that Cohort 2 respondents reported significantly more psychi- 
atric symptoms one year after the initial interview than did respondents 
in Cohort 1 (¢ = 2.40, df = 524; P< .01), primarily in Cincinnati and 
Toledo. The 2-month follow-up symptom levels did not differ signifi- 
cantly between the cohorts (table 3). 

Although none of the city-by-cohort interactions at the 12-month fol- 


ca 


Continutty of Care and Chent Outcomes 115 


TABLE 3 
Client Clinical Outcomes: City x Cohort-adjusted Means 
Mean? differences and probabilities 
Two months after baseline Twelve months after baseline 
City Cohort 1 Cohort 2 Difference Cohort 1 Cohort 2 Difference 


Average number of nights 
hospitalized per month 


Baltimore 56 .70 0.14 3.03 2.78 —0.25 
Cincinnati .40 41 0.01 1.78 2.56 0.78 
Columbus 39 .29 —0.10 1.74 1.25 —0.49 
Toledo 27 18 —0.09 2.56 1.31 —1.25 
All cities 41 49 —0.02 2.28 1.97 —0.31 
SCL-90° 

Baltimore 2.16 2.04 —0.12 2.03 2417 0.14 
Cincinnati 2.04 2.23 0.19* 1.97 2:22 0.25** 
Columbus 2.04 2.05 0.01 2.09 2.00 —0.09 
Toledo 2.03 1.89 —0.14 1.31 1.69 0.38* 
All cities 2.07 2.05 —0.02 1.85 2.02 0.17** 


4 Adjusted for baseline lagged variable and subject characteristics. 
b1 = not at all, 4 = quite a lot. 
Pie 05; F S01, 


low-up achieved significance, modest interaction effects were observed 
for general life satisfaction and satisfaction with living situation 2 months 
after discharge. However, an inspection of the adjusted means from both 
cohorts in each city revealed that Toledo produced significant differences 
favoring Cohort 2 for both of these variables (see table 4). A significant 
city X cohort interaction was observed for respondents’ self-rated level of 
functioning (F = 2.86, df = 3, 566; P< .05) 2 months after discharge. 
This finding is apparently due to respondents in Cincinnati Cohort 1 rat- 
ing their own functioning better than did those in Cincinnati Cohort 2 
(¢ = 2.17; P< .05). 


Discussion 


The first major finding of this study is that improvements in services of 
the types targeted by the RWJF program —specifically, enhancing the 
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TABLE 4 
Client Outcomes: City x Cohort-adjusted Means 


Mean? differences and probabilities 


Two months after baseline Twelve months after baseline 
City Cohort 1 Cohort 2 Difference Cohort 1 Cohort 2 Difference 
General life satisfaction? 
Baltimore 4.64 4.35 —0.29 4.44 4.22 —0.22 
Cincinnati 4.63 4.43 —0.20 4.70 4.61 —0.09 
Columbus 4.33 4.39 0.06 4.55 4.62 0.07 
Toledo 4.82 5.46 0.64* 5.52 5.73 0.21 
All cities 4.61 4.66 0.05 4.80 4.79 —0.01 
Satisfaction with living situation? 
Baltimore 4.96 4.93 —0.03 5.10 4.89 —0.21 
Cincinnati 5.19 4.85 —0.34 5.04 5.12 0.08 
Columbus 4.92 5.13 0.21 5.13 V1; 0.04 
Toledo 5.41 6.01 0.60* 6.70 5.09 —1.61* 
All cities 5.12 5.23 0.11 5.49 527 —0.22 
Self-rated level of functioning‘ 
Baltimore 2.56 2.34 —0.22 2.27 2.72 0.45** 
Cincinnati 2.31 2.62 0.31* 2.34 2.39 0.05 
Columbus 2.47 2.31 —0.16 2.28 2.45 0.17 
Toledo 2.32 2.33 0.01 2.08 1.99 —0.09 
All cities 2.41 2.40 —0.01 2.24 2.39 0.15 


* Adjusted for baseline lagged variable and subject characteristics. 
>1 = terrible, 7 = delighted. 

“1 = excellent, 4 = poor. 

*P< .05, ** P< .O1. 


provision and continuity of case management — were found for individ- 
ual clients in three out of the four cities studied. The second, and less 
encouraging, major finding is that we were unable to detect improve- 
ments in client outcomes related to these enhancements in case manage- 
ment or to the program as a whole. In fact, psychiatric symptoms were 
statistically, although probably not clinically, significantly worse in 
Cohort 2 compared with Cohort 1 in two of the cities. (The overall 
between-cohort difference on symptoms is 0.20 standard deviation, a 
small effect.) Each of the findings deserves further consideration to un- 
derstand its significance. 
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The finding for expansion of case management services reflects the 
success of the local mental health authorities in the three Ohio cities in 
carrying out this targeted system change. These three local authorities 
believed that the route to improving continuity of care for clients was 
through case management, and the data indicate that they achieved this 
goal. Their success suggests that the authorities were able to direct re- 
sources to effect desired changes (Roth et al. 1993). Clearly, such 
changes indicate the intended capacity of a local mental health authority 
to assume a systemwide planning role and to implement change through 
control of financial streams (Aiken, Somers, and Shore 1986; Lehman 
1989; see also the article by Morrissey et al. in this issue). 

The finding of no increase in case management services in Baltimore 
illustrates both some of the pitfalls in conducting this type of evaluation 
and the variability in the process of implementing change through local 
mental health authorities. As Morrissey et al. discuss elsewhere in this 
issue, Baltimore was slower than the Ohio cities in implementing the 
activities of the local authority, but authority functioning also improved 
rapidly there during the last year of the project. The timing of this client 
outcome study was such that any changes in the quantity and continuity 
of case management services in Baltimore may have occurred after the 
client data were collected. A further complication in Baltimore, unfore- 
seen at the time this study was being conducted, was that the areas of 
the city from which the majority of the client sample was drawn expert- 
enced the greatest delay in implementating new case management ser- 
vices. This delay stemmed largely from the politics of establishing in 
these areas the “lead agency” through which the case management ser- 
vices were performed. This unfortunate convergence of delayed fulfill- 
ment of services in the very areas of the city where the evaluation was 
conducted arose partly from happenstance and partly from the fact that 
the politically most complicated area of the city also housed the major 
state psychiatric facility from which clients were recruited. It should be 
noted, however, that the continuity of case management, as measured 
by the lack of change in case managers, improved significantly in Balti- 
more in Cohort 2. 

Despite this variability in achieving case management across the cities, 
it is nonetheless impressive that the service effects of the program carried 
out at the level of a citywide authority were clearly detectable in a pre- 
dictable manner at the client level. Such service penetration indicates a 
definite impact of the program on services throughout the system. 
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The positive findings regarding case management are offset by the 
negative findings about outcomes; that is, there was no detectable im- 
provement in outcomes between the two cohorts in any of the cities 
across multiple domains. This finding requires consideration of both the 
methodological limitations of the study and the realities of changing 
outcomes for persons with CMI. 

One evaluation challenge was the fact that the “program” —the cte- 
ation of the local mental health authority — assumed different forms and 
was instituted with different degrees of success and rapidity across the 
cities (see Morrissey et al. in this issue). Thus, the intervention being 
evaluated was heterogeneous across the sites, and even across areas 
within sites, as we have discussed with reference to Baltimore. This het- 
erogeneity complicated detection of service impact and would tend to 
obscure more specific effects. Much more detail than our evaluation sup- 
plied regarding the nature and intensity of services received by individ- 
ual clients would be needed to investigate fully the relation of service 
heterogeneity to client outcomes. 

A second methodological issue concerns the timing and length of the 
follow-ups. We chose the 2-month and 12-month follow-up points in 
order to capture any short-term postdischarge effects as well as longer- 
term effects. We did, in fact, note considerable postdischarge improve- 
ment in clients between baseline and the 2-month follow-up (not reported 
in this article). What we failed to find was differential improvement be- 
tween the cohorts, that is, a program effect. Clients showed the most 
dramatic improvements during these first 2 months after discharge and 
then leveled off between 2 and 12 months. A major criticism could be 
made that the follow-up period was not long enough. It is frequently 
noted that outcome improvements for persons with CMI often require 
considerable time — more than a year. Hence, it can be argued that the 
follow-up time was insufficient to detect improvements. We were tfe- 
stricted to the one-year follow-up because of the pre—post cohort design 
and budget. The companion study by Shern et al. in this issue is helpful 
because they were able to follow cohorts in the two cities over a five-year 
period, using their comparison-site design. The similarity of their find- 
ings to those in this study tends to argue against an inadequate follow-up 
period as the major reason for failure to find outcome effects. 

Third, the timing of the overall evaluation may have been problem- 
atic. Funding for establishing the local mental health authorities began 
in November 1986, a full year before recruitment of the “baseline” co- 
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horts (Cohort 1) began. Therefore, it is possible that the evaluation 
missed early improvements in the treatment system related to the RWJF 
program. The site-level evaluation conducted by Morrissey and col- 
leagues indicates that this may have been the case in the Ohio cities. 
Conversely, as previously noted, in Baltimore, where implementation 
was slow, the timing of Cohort 2 may have occurred too early to detect 
effects of the program at the client level. 

Even with these caveats about the evaluation design, the finding of an 
overall increase in case management without improvements in outcomes 
is disappointing. The research results elsewhere on the outcomes of case 
management are variable (Attkisson et al. 1992). Some studies report 
convincing benefits of case management (Goering et al. 1988: Borland, 
McRae, and Lyca 1989; Olfson 1990), whereas others report no benefits 
or even disadvantages, such as increased service consumption without 
outcome improvement (Franklin et al. 1987). The major problem with 
this body of research, of which this article can be viewed as another ex- 
ample, is that “case management” differs considerably across studies and 
is often poorly defined. 

A second problem is that case management is often applied indiscrim- 
inately to all clients with the expectation that it is a panacea. Such indis- 
cfiminate application of the intervention will obscure its differential 
impact for client subgroups. In addition, the effectiveness of case man- 
agement within a system of care is undoubtedly affected by the quality 
of other available services. An excellent case management service may be 
able to accomplish little if the rest of the service system is grossly inade- 
quate. In this study we have essentially no information on the quality of 
the case management offered across the sites or differentially to clients 
within sites. Reflecting the logic of the program itself, the method of 
analysis here assumes that case management in itself is an appropriate 
intervention for all persons with CMI. Our analytic approach ignores 
issues of appropriate matching of clients to case management. 

Finally, in the RWJF cities studied, the major service change was more 
case Management, a premise supported by data from the Ohio Depart- 
ment of Mental Health, although in some cities there were increases 
in certain other services (e.g., medical services in Cincinnati and outpa- 
tient counseling services in Columbus) (Roth et al. 1993). It may be that 
although increased case management was a necessary first step toward 
improving outcomes, it was not sufficient. Perhaps other community 
services should be developed or enhanced to realize the benefits of better 
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case management. As we feport, the provision of certain other catego- 
ries of services actually decreased in some cities over the course of the 
demonstration. 

Clearly the relation between change in services and outcomes ts com- 
plex. The most judicious interpretation of the findings in this client 
study may be that the steps taken to create local mental health authotri- 
ties and to improve case management and continuity of care in the cities 
were necessary, but not sufficient, steps toward improving client out- 
comes. More attention should be given to the quality and quantity of 
available services other than case management. There is substantial evi- 
dence from well-controlled efficacy studies that high-quality clinical, re- 
habilitative, and supportive services (Attkisson et al. 1992; Hargreaves 
and Shumway 1989), including pharmacotherapies (Kane and Lieber- 
man 1987), family educational programs (McFarlane 1983; Anderson, 
Reiss, and Hogarty 1986), skills training (Liberman, Mueser, and Wal- 
lace 1986; Benton and Harold 1990), and assertive community treatment 
teams (Stein and Test 1980; Olfson 1990), among others, can improve 
client outcomes. The challenge remains to disseminate these approaches 
and technologies effectively so that all persons with CMI can benefit 
from them. 


References 


Aiken, L.H., S.A. Somers, and M.F. Shore. 1986. Private Foundations 
in Health Affairs: A Case Study of the Development of a National 
Initiative for the Chronically Mentally Ill. Amzerican Psychologsst 
41:1290-5. 

Anderson, C.M., D.J. Reiss, and G.E. Hogarty. 1986. Schizophrenwu 
and the Family. New York: Guilford Press. 

Attkisson, C., J. Cook, M. Karno, et al. 1992. Clinical Services Research. 
Schizophrenia Bulletin 18:561-625. 

Bachrach, L.I. 1981. Continuity of Care for Chronic Mental Patients: A 
Conceptual Analysis. American Journal of Psychiatry 138:1449-56. 

Benton, M.K., and H.E. Harold. 1990. Social Skills Training with 
Schizophrenics: A Meta-Analytic Approach. Journal of Consulting 
and Clinical Psychology 58:741-7. 

Borland, A., J. McRae, and C. Lyca. 1989. Outcomes of Five Years of 
Continuous Intensive Case Management. Hospital and Community 
Psychiatry 40:369-76. 


Continuity of Care and Chent Outcomes 121 


Ciarlo, J.A., and J. Rehman. 1977. The Denver Community Mental 
Health Questionnaire: Development of a Multidimensional Pro- 
gram Evaluation Instrument. In Program Evaluation for Mental 
Health: Methods, Strategies, and Participants, eds. R. Coursey et al. 
New York: Grune and Stratton. 


Derogatis, L. 1983. SCL-90: Administration, Scoring, and Procedures 
Manual-II. Baltimore: Clinical Psychometric Research. 

Franklin, J.L., B. Solovitz, M. Mason, J. Clemons, and G. Miller. 1987. 
An Evaluation of Case Management. American Journal of Public 
Health 77:674-8. 

Goering, P., D. Wasylenki, M. Farkas, W. Lance, and R. Ballantyne. 
1988. What Difference Does Case Management Make? Hospital and 
Community Psychiatry 39:272-6. 

Goldstrom, I.D., and R.W. Manderscheid. 1986. The Chronically Men- 
tally Ill: A Descriptive Analysis from the Uniform Client Data In- 
strument. Community Support Service Journal 2:4-9. 

Hargreaves, W.A., and M. Shumway. 1989. Effectiveness of Mental 
Health Services for the Severely Mentally Ill. In The Future of Men- 
tal Health Services Research, eds. C.A. Taube, D.A. Mechanic, and 
A.A. Hohmann, 253-83. DHHS pub. no. (ADM) 89-1600. Wash- 
ington. 

Kane, J.M., and J.A. Lieberman. 1987. Maintenance Pharmacotherapy 
in Schizophrenia. In Psychopharmacology: The Third Generation of 
Progress, ed. H.Y. Meltzer, 1101-9. New York: Raven Press. 

Lehman, A.F 1988. A Quality of Life Interview for the Chronically 
Mentally Ill. Evaluation and Program Planning 11:51-62. 

. 1989. Strategies for Improving Services for the Chronic Men- 
tally Ill. Hospital and Community Psychiatry 40:916-90. 

Lehman A.F., N. Ward, and L. Linn. 1982. Chronic Mental Patients: 
The Quality of Life Issue. Amzerican Journal of Psychiatry 10:1271-6. 

Liberman, R.P., K. Mueser, and C.J. Wallace. 1986. Social Skills Train- 
ing for Schizophrenic Individuals at Risk for Relapse. American 
Journal of Psychiatry 143:523-6. 

McFarlane, W.R. Ed. 1983. Family Therapy in Schizophrenia. New 
York: Guilford Press. 

Olfson, M. 1990. Assertive Community Treatment: An Evaluation of 
the Experimental Evidence. Hospital and Community Psychiatry 
41:634-41. 

Roth, D., J.A. Clark, E.G. Lauber, and M.F. Hogan. 1993. Two Views 
of a Service System: Client Movement and Client Needs. In Pro- 
ceedings of the Third Annual Conference on State Mental Health 
Agency Services Research, 147-64. Alexandria, Va.: NASHPD Re- 
search Institute. 


122 Anthony F. Lehman et al. 


Shore, M.F, and M.D. Cohen. 1990. The Robert Wood Johnson Foun- 
dation Program on Chronic Mental Illness: An Overview. Hospital 
and Community Psychiatry 41:1212-16. 

Stein L.I., and M.A. Test. 1980. Alternative to Mental Hospital Treat- 
ment. I: Conceptual Model, Treatment Program, and Clinical Eval- 
uation. Archives of General Psychiatry 37:392-7. 


Tessler, R.C., G. Willis, and G.D. Gubman. 1986. Defining and Mea- 
suring Continuity of Care. Psychosocial Rehabilitation Journal 
10:27-38. 


Coauthors of the article are: LETICIA T. POSTRADO, University of Maryland 
School of Medicine; DEE ROTH, Ohio Department of Mental Health; Scot W. 
MCNARY, University of Maryland School of Medicine; and HOWARD H. GOLD- 
MAN, University of Maryland School of Medicine. 


Address correspondence to: Anthony F. Lehman, MD, Center for Mental Health 


Services Research, University of Maryland School of Medicine, 645 Redwood 
Street, Baltimore, MD 21201. 


